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Su pplemental Action 

Errata: The Final Office Action of February 24, 2005 

1 . On March 24, 2005, the Applicant's representative, Carole Quinn (Registration No. 39000), pointed out 
that the Final Office Action, filed February 24, 2005 (hereinafter, the Final Office Action), lacked a treatment of 
Claim 12. The purpose of this communication is to address those and other errors found in the Final Office Action. 

2. The Applicant will notice that in the Final Office most dependant claims were rejected under the same 
grounds as presented in the original Office Action (filed, April 6, 2004; hereinafter, the Original Office Action). An 
explicit statement of rejection was, however, accidentally omitted for Claim 12. The rejection of Claim 12 is 
provided below. 

3. Other errors in the Final Office Action include: 

a. On page 4, % 7 of the Final Office Action, no acknowledgement was made to the Applicant's 
cancellation of Claim 1 9. Paragraph 7 on page 4 of the Final Office Action should be revised to: 

The amendments to the Claims, filed October 12, 2004, have been made of record. Claims 1, 2, and 13 have 
been amended accordingly. Claim 19 has been cancelled. 

b. Because Claim 19 is cancelled, the statements of rejection should not have stated Claim 19 as 

being rejected. Therefore, the following paragraphs should be revised as follows. 

Paragraph 9 on page 4 of the Final Office Action should read: 

Claims 1-3, 5-7, 9-10, 17-18, 20-24, and 31 remain rejected under 35 U.S.C. § 102(b) as being anticipated by 
[ChangSmith95] (S. Chang and J.R. Smith, Extracting Multi-Dimensional Signal Features for Content-Based 
Visual Query, SPIE Symposium on Visual Communications and Signal Processing, May 1995). 

Similarly, paragraph 34 on page 1 1 should be read: 

Claims 1-3, 5-7, 9-10, 17-18, 20-24, and31 are rejected under 35 U.S.C. § 102(b) as being anticipated by 
[ChangSmith95]. 

Paragraph 37 on page 13 should be read: 

The following is in regard to Claims 3, 5-7, 9-10, 1 7-/8, 20-24, and 31. The rejections of Claims 3, 5-7, 9-10, 
1 7-24, and 31 set forth in the previous Office Action are incorporated, herein, by reference. For the sake of 
brevity, the details of these rejections are omitted. Please refer to the appropriate sections of the previous Office 
Action, as well as the discussion of [ChangSmith95] above. 
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c. In paragraphs 13 and 45, Claim 31 was accidentally indicated as being allowable. This claim was 
clearly rejected in both the Final Office Action and the Original Office Action. Claim 30 was 
considered allowable, not Claim 31. Therefore, paragraph 13 on page 4 of the Final Office Action 
should be read as: 

Claims 4, 8, 26, and 30 contain allowable subject matter, but are dependant on the rejected claims above. 

Similarly, paragraph 45 on page 15 of the Final Office Action should be read as: 

Claims 4, 8, 26, and 30 are objected to as being dependent upon a rejected base claim, but would be allowable if 
rewritten in independent form including all of the limitations of the base claim and any intervening claims. 

d. Paragraph 10 on page 4 of the Final Office Action should read as: 

Claims 11,12, 14, 25, and 27-29 remain rejected under 35 US.C. § 1 03(a) as being an obvious modification of 
[ChangSmith95], in view of [SmithChang94] (J.R. Smith and S. Chang, Quad-Tree Segmentation for Texture- 
Based Image Query, ACM 2 nd International Conference on Multimedia, October 1994). 

Other changes are shown in the corrected version of the Final Office Action. See below. 
Rejections Under 35 U.S.C. $ 103fa): Claim 12 

4. The following is a quotation of 35 U.S.C. § 103(a) which forms the basis for all obviousness rejections set 
forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if 
the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would 
have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claim 12 is rejected under 35 U.S.C. § 103(a) as being unpatentable over [ChangSmith95] (S. Chang and 
J.R. Smith, Extracting Multi-Dimensional Signal Features for Content-Based Visual Query, SPIE Symposium on 
Visual Communications and Signal Processing, May 1995), in view of [SmithChang94] (J.R. Smith and S. Chang, 
Quad-Tree Segmentation for Texture-Based Image Query, ACM 2 nd International Conference on Multimedia, 
October 1994). 

6. The following is in regard to Claim 12. The basis of the rejection of Claim 12 was provided on page 10, U 
24 of the Original Office Action. For the sake of clarity this will be reiterated. The teachings of [ChangSmith95] can 
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be combined with those of [SmithChang94] in such a way as to satisfy all the limitations of Claim 1 1 . The step of 
calculating the first similarity in the method of [SmithChang94] is implemented by calculating a distance (e.g. 
Mahalanobis distance between textures features - Section 4. 1 on page 4 of [SmithChang94]) between the second 
information item associated with example image and the second information item associated with the said stored 
image under consideration. In this way, the teachings of [ChangSmith95] and [SmithChang94], when combined in 
the manner described above, yield a method of searching for image that conforms to all the limitations of Claim 12. 

7. A corrected version of the Final Office Action, which shows these changes, is provided in the appendix on 
the following page. 
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Appendix: Corrected Final Office Action of February 24, 2005 

Detailed Action 
Drawings 

Response to Drawing Corrections 

1. New and corrected drawings were received on October 12, 2004. The replacement drawings are 
unacceptable. 

2. As pointed out in the previous Office Action (previous Office Action, page 2, paragraph 2), steps S609 and 
S623 are ambiguous because the Applicant has failed to adequately provide a definition of function^ ). The 
Applicant has indicated in Remarks, filed October 12, 2004, that the definition of function{ ) corresponds to the 
discussion found on page 23, line 1 1 of the originally filed Specification. This correspondence, however, is not 
readily apparent. On page 23, line 1 1, the following expression can be found: 

Match(Q i D S ) = f j d(h?,h? s ) (3) 

This expression is related only to the process shown in Fig. 6b. Technically speaking, the right hand side of equation 
(3) is a function of the feature vector elements, hf and hf $ i = 1 ...Af, whereas fitnction( ) is function of the feature 
vectors (i.e. H(Q) and H(Ds)) themselves. This, however, is a minor detail. The main concern with using the 
undefined function function^) is that it gives the impression that Match^Q^Ds), or the result thereof, is an arbitrary 
function of the feature vectors H(Q) and H(Ds). According to the Applicant's disclosure, this is clearly not the case. 
At the minimum,yi//7C//0/?( ) must indicate the similarity between the vectors H(Q) and H{Ds). In and of itself, the 
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function functionQ does not indicate this. Furthermore, the Applicant should notice that functionQ is used in step 
S609 of Fig. 6a and in step S623 of Fig. 6b. This could imply that the functions used to evaluate Filter(Q,Ds) and 
Match(Q,Ds) in steps S609 and S623 are the same. However, according to the Applicant's disclosure, different 
metrics of similarity are used in steps S609 and S623. 

3. These issues can be overcome by replacing functionQ in step S609 with something that indicates a first 
similarity (cf. page 21, lines 3-6 of the Specification) - e.g. first similarity ) - and replacing functionQ in step 
S623 with something that indicates and a second similarity (cf. page 22, lines 25-30) - e.g. second_similarity{ ); or 
by substituting functionQ with the disclosed mathematical expressions indicating similarity (e.g. equation (3) 
above). 

4. The objections posed in the previous Office Action concerning these issues are, therefore, maintained. All 
other objections are withdrawn. 

5. Corrected drawing sheets are required in reply to the Office action to avoid abandonment of the application. 
Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version 
of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should 
not be labeled as "amended." If a drawing figure is to be canceled, the appropriate figure must be removed from the 
replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made 
to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be 
necessary to show the renumbering of the remaining figures. The replacement sheet(s) should be labeled 
"Replacement Sheet" in the page header (as per 37 CFR 1 .84(c)) so as not to obstruct any portion of the drawing 
figures. If the changes are not accepted by the examiner, the applicant will be notified and informed of any required 
corrective action in the next Office action. The objection to the drawings will not be held in abeyance. 
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Specification 

Response to Amendments to the Specification 

6. The amendments to the Specification, filed October 12, 2004, have been acknowledged. These amendments 
correspond to acceptable changed made to the drawings. 

Claims 

Response to Amendments to the Claims 

7. The amendments to the Claims, filed October 12, 2004, have been made of record. Claims 1, 2, and 13 
have been amended accordingly. Claim 19 has been cancelled. 

8. As a result of these amendments, the prior rejection of Claim 13 under 35 U.S.C. § 1 12(2) is withdrawn. 

9. [Claims 1-3, 5-7, 9-10, 17-24, and 311 Claims 1-3,5-7, 9-10, 17-18,20-24, and 31 remain rejected under 35 
U.S.C. § 102(b) as being anticipated by [ChangSmith95] (S. Chang and J.R. Smith, Extracting Multi- Dimensional 
Signal Features for Content-Based Visual Query, SPIE Symposium on Visual Communications and Signal 
Processing, May 1995). 

10. Claims 1 lj^ 14, 25, and 27-29 remain rejected under 35 U.S.C. § 103(a) as being an obvious 
modification of [ChangSmith95], in view of [SmithChang94] (J.R. Smith and S. Chang, Quad-Tree Segmentation 
for Texture-Based Image Query, ACM 2 nd International Conference on Multimedia, October 1994). 

11. Claim 13 remains rejected under 35 U.S.C. § 103(a) as being an obvious modification of [ChangSmith95], 
in view of [FolkersOO] (A. Folkers, Pictorial Query Specification and Processing, Ph.D. Thesis - University of 
Maryland and Medical University of Lttbeck, January 2000). 
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12. Claims 15-16 remain rejected under 35 U.S.C. § 103(a) as being an obvious modification of 
[ChangSmith95], in view of [Smith97] (J.R. Smith, Integrated Spatial and Feature Image Systems: Retrieval 
Analysis and Compression, Ph.D. Thesis - Columbia University, 1997). 

13. Claims 4, 8, 26, and £4-30 contain allowable subject matter, but are dependant on the rejected claims 
above. 

Response to Arguments and Remarks 

1 4. Before proceeding, the Applicant is informed that an explicit statement of rejection was accidentally 
omitted in the previous Office Action for Claims 10 and 17-24. The Applicant will notice, however, that grounds for 
the rejections of these Claims were provided. Please refer to paragraphs 16-17 on pages 6-7 of the previous Office 
Action. 

15. A brief summary of the Applicant's invention is presented, followed by an overview of [ChangSmith95]. In 
the discussion of [ChangSmith95], the Applicant's arguments concerning the "degree of significance of the visual 
content of at least one block of the image with respect to the overall visual content" will be addressed (pages 17-18 
of the Applicant's Remarks, filed October 12, 2004). A brief discussion of [SmithChang94], [Smith97], and 
[FolkersOO] will follow, illustrating the manner in which these references compensate for the deficiencies of 
[ChangSmith95]. 

1 6. The Applicant 's Invention. The Applicant's invention relates generally to a method of indexing in a content 
based image querying (CBIR) system. Specifically, according the "Applicant's methodology, indices IDXarc 
generated for a given query image, g, and each of the database images, D (cf. Fig. 6a). Each of the indices consist of 
two elements: the first sub-index, H, "characteristic of the visual content of the image", and the second sub-index, 
W, "characteristic of the spatial distribution of the visual content" of the image. The database is parsed in search of 
database images, D, that match the query image, Q, by first evaluating the similarity of their respective second sub- 
indices, W(D) and W(Q) t with respect to some metric of similarity (e.g. FilteriQJD) - cf. Fig. 6a). This process 
produces a set of database images Ds. This set is then parsed by evaluating the degree of similarity between the first 
sub-index H(Q) of the query image and the first sub-indices W(Ds) of the previously obtained database images. The 
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Ds that satisfy the given criterion of similarity (as defined by, e.g., Match(Q t D) - cf. Fig. 6b) should represent the 
set of images in the database which most closely match the query image Q. 

1 7. According to the Applicant's method, in order to generate an index for a given image Im, the image is first 
decomposed into //blocks. This decomposition may either be a regular decomposition or quad-tree decomposition. 
In accordance with the former, Im is partitioned into N regularly ordered blocks, all of the same fixed-sized. In quad- 
tree decomposition, the image is recursively sub-divided into groups of four successively smaller blocks. Although 
not disclosed by the Applicant, each subsequent phase of the decomposition typically results in blocks exhibiting 
more homogenous content than the larger blocks of preceding phases. Regardless of the method of decomposition, 
the Applicant associates a first data item h) m with each of the N blocks. h) m is indicative of the visual content with 
the /-th block of the image and, in the case of the Applicant's method, is represented as a histogram. Collectively, 
these first data items are indicative of the visual content of the entire image. This concept is embodied in the 
Applicant's first sub-index H(Im\ which is an TV-dimensional feature vector (tensor) having, as elements, all TV block 

histograms hj m - i.e. H(Im) - (^ m ) M . Each block / of the decomposed image is also associated with a second 
data item w] m , which is indicative of the "visual significance" of the block with respect to the overall visual content 
of the image 1 . The data items w, Im indicate the "spatial distribution of the visual content" of the image Im, and can be 
collectively expressed as the second sub-index W(Im) - i.e. W(Im) = (w/ m ) - Together, the sub-indices H(Im) 
and W{Im) form the index IDX(Im) of the image Im. 

1 8. The Applicant proposes several methods for evaluating the similarity between the sub-indices of two 
images. For the first sub-index H, the Applicant suggests using well-known metrics, such as histogram intersections 

Notice that j/r, 1 " J 1 is the energy of the histogram of block i. In keeping with this convention, the total energy associated with the image is 

SvJI^ir - Therefore, the ratio U ' '1/^,11 • II signifies me fraction of the total histogram energy per block /. Given the similarity of 
this expression with Applicant's equation (1), it can be inferred that w* likewise provides a measure of the (relative) amount of energy 
deposition per block. [Smith97], for example, shows the relationship between histogram energy (or, more precisely, the fraction of total 
histogram energy) and visual significance (cf. [Smith97] Fig. 2-4 and Section 4.5.3). Although the claims related to were deemed 
allowable, the Prior Art may provide some motivation for measuring the relative block energy, as essentially does. 
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or the Euclidean distance between histograms. For the second sub-index W, the Applicant suggests the Euclidean 
distance between the vectors W(Q) and W(D\ in cases where the image has been regularly decomposed. If, on the 
other hand, a quad-tree decomposition is used, the Applicant suggests evaluating similarity on the basis of the 
isomorphism between the quad-tree of the query image and that of the database images. Although not particularly 
appealing in terms of computational complexity, this method has been utilized by other authors in CBIR or image 
comparison applications. 

1 9. In summation, Applicant's method generally involves indexing a database of images - first according to the 
spatial distribution of visual content throughout an image, as represented by the image's second sub-index Wand by 
the image's quad-tree structure; and, secondly, in accordance with the global visual content of the image, as 
represented by the first sub-index H. The Prior Art CBIR methods cited below generally involve an evaluation of 
both the spatial distribution of the visual content within an image and the global visual characteristics associated 
with the image. 

20. [ChangSmith95]. [ChangSmith95] studies the extraction of multi-dimensional features (e.g. color, texture 
shape, etc.) from images for the purposes of CBIR. These features can be used for indexing a database of images. 
[ChangSmith95] is concerned primarily with extracting visual features corresponding to texture, color, and shape 
(only the aspects of [ChangSmith95] relating to texture and color are discussed here, as they are most pertinent to 
the Applicant's claimed invention). These aspects of an image essentially define the visual content of an image. 

21 . Using a spatial quad-tree approach, the image is segmented into sub-regions exhibiting substantially 
homogenous texture ([ChangSmith95], page P-5). Textures are analyzed in the wavelet domain. Features are derived 
for each of block of the decomposed image. These include the subband energy(s) a) of each block, (cf. 
[ChangSmith95] Fig. 2) 2 . The texture features of all blocks form a set of features that may be used to index the 
visual content of the image (cf. [ChangSmith95] page P-5, second to last paragraph). 



2 



Note that the wavelet transform results in a decomposition of the block into several subbands, as depicted in [ChangSmith95] Fig. 2. 
Therefore, the derived texture features for each block are not limited to a single subband energy. 
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22. Beginning with the same quad-tree structure employed during the texture analysis, [ChangSmith95] derives 
the histogram for each terminal node (block) of the quad-tree. In a manner similar to the texture analysis above, the 
resulting quad-tree decomposition consists of blocks having substantially similar colors. The color histograms, much 
like the texture features above, collectively form a set of color features which may be used to index the visual 
content (color) of the image ([ChangSmith95], Section 2.2). 

23. The disparate visual features of shape, color, and texture are integrated into a single data structure called an 
integrated feature map ([ChangSmith95], Section 3). The integrated feature map is a quad-tree, whereon the texture, 
color, and shape features are mapped. This data structure allows an image to be indexed according to a variety of 
different features (e.g. texture, color, or shape). 

24. The integrated feature map (IFM) of [ChangSmith95], therefore, serves as an image index, whereby the 
visual content of an image is expressed in terms of at least two sets of features - e.g. texture and color. Since each of 
the constituent feature sets is compared independently during the querying process, each feature set can be treated as 
a "sub-index" of an image. These indices could arbitrarily be denoted as the "first" sub-index, "second" sub-index, 
and so on. However, in order to remain consistent with the Applicant's nomenclature (wherein the first sub-index 
takes into account the visual content of the image, and the second sub-index is "indicative of a degree of 
significance of the visual content ... with respect to the overall content of the image" and "characteristic of the 
spatial distribution of the visual content of the image"), the set of color features will be treated as the first sub-index 
and the set of texture features will be treated as the second sub- index 3 . 

25. As stated above, the set of color features utilized in [ChangSmith95] consists the color histograms 
calculated at each terminal node, or leaf, of the quad-tree ([ChangSmith95], Section 2.2, Quad-Tree Based Color 
Histogram Indexing, paragraph 1). This set of features is essentially identical, both in form and purpose, to the 
Applicant's first sub-index H, which is the set of histograms calculated for all blocks of the decomposed image (cf. 
Applicant's Specification, page 6 last paragraph and page 15, line 24 to page 16, line 4). Like //, the set of color 
histograms describes the overall color content of an image. Note that, as a result of the quad-tree decomposition, the 



3 



Notice that the first sub-index and the second sub-index correspond, respectively, to the "first information item" and the "second information 
item" of the Claims. The corresponding terms and will be used interchangeably throughout this document. 
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set of histograms are distributed in a spatial ordering. As such, the set of histograms is also indicative of the spatial 
distribution of visual (color) content. 

26. In [ChangSmith95], the set of texture features include the subband energy, or wavelet energy, for each leaf 
of the quad-tree decomposed image (cf. [ChangSmith95] Fig. 2 and Section 2.1, Modified Quad-Tree Based Texture 
Region Segmentation). [ChangSmith95] shows that distinctive regions of texture (i.e. "visually significant" regions) 
can be characterized by their respective wavelet energy signatures. These regions are located at the leaves of the 
quad-tree decomposition of the image (cf. [ChangSmith95] Fig. 4 and Section 2. 1, Modified Quad-Tree Based 
Texture Region Segmentation). Again, each of these leaves corresponds to a block of an image exhibiting locally 
homogeneous texture. Seen in this light, the wavelet energy features represent a "data item of a second type [wavelet 
energy] indicative of the degree of visually significance [distinctiveness] of the visual content" (i.e. indicative of the 
texture) "of a block under consideration" (i.e. a leaf of the quad-tree decomposition of the image). Further note that 
the distinctiveness ([ChangSmith95], Section 2. 1 , paragraph 1) of the block texture can only be differentiated by 
observing the block with respect to the texture characteristics of entire image (i.e. "with respect to the overall visual 
content of the image"). Thus, the set of wavelet energies for each block (this set being represented by the quad-tree) 
collectively represent a "second information item", or second sub-index, "indicative of a degree of significance of 
visual content of at least one of the plurality of blocks [of the quad-tree decomposition of the image], with respect to 
the overall content of the image". Again, the spatial 'distribution of these features (which are indicative of the visual 
content of the image) throughout the image is given by the spatial quad-tree decomposition discussed above. 

27. These wavelet energy signatures also reflect the distribution of energy in the spatial-frequency domain for 
each block of the decomposition. Therefore, the wavelet energy signature of a block provides insight into the block's 
spatial-frequency characteristics. For example, blocks having wavelet energy distributed primarily at the low end of 
the spatial-frequency spectrum correspond to flat, uniform image regions, whereas blocks with energy deposited in 
higher frequency subbands correspond to textures or edges. In this sense, the wavelet energy signatures of a block 
again provide a measure of "a degree of significance of visual content . . . with respect to the overall content of the 
image", depending on what is defined to be significant (e.g. flat regions or textured regions). Clearly, in the texture 
segmentation and extraction phase of [ChangSmith95], texture is of particular significance. From this perspective, it 
can once again be concluded that the set of all wavelet energy signatures represents a "second information item" in 
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accordance with Claim 1. Further support for this proposition can be found in [SmithChTexture94], 
[VanDeWouwer97], [ChangKuo93], and [LiangKuo99]. 

28. [SmithChang94]. [SmithChang94] provides a more extensive treatment of the spatial quad-tree 
decomposition discussed above. Furthermore, [SmithChang94] proposes the concept of hierarchically searching the 
image database ([SmithChang94], Section 6.1). Specifically, with hierarchical searching, each dimension of the 
feature space is searched independently in accordance with the significance of each dimension. As each dimension is 
searched, the matching set of images is successively reduced until some desired number of retrievals has been 
obtained. However, in [ChangSmith95], and CBIQ applications in general, the dimensionality of the feature space 
may be prohibitively large, and would thereby render the hierarchical search of every dimension impractical. An 
obvious alternative to searching the entire feature space of [ChangSmith95] would be to hierarchically search 
according to color and texture - essentially treating the set of texture-based features and the set of color-based 
features as separate dimensions. This observation would have been apparent to one of ordinary skill in the art. The 
question of which set of features (i.e. which dimension) is most significant was addressed in the previous Office 
Action (previous Office Action, pages 8-9). To expand on that discussion, notice that the significance of texture- 
based features over color-based features follows intuitively from one's own perceptual experience. For example, an 
orange is visually distinguishable from a like-sized orange sphere because of its surface texture. This would be the 
case even if the color were removed. The dominance of texture-based features over color-based features in human 
visual perception has been extensively verified in studies of the human visual perception system. 

29. [ChangSmith95] also supports this decision of significance being made according to the subjective 
judgment of a user, as represented by his/her submitted query (cf. [ChangSmith95] page 9, paragraphs 1-2). 

30. [Smith97]. [Smith97] gives a thorough description of the technological and theoretical underpinnings of 
Columbia University's VisualSEEk CBIR system. [ChangSmith95] and [SmithChang94] are also related to this 
system. In relation to the Claims, [Smith97] teaches a "sum of the distances [- i.e. a Minkowski distance metric -] 
between each of the components of [an] information item associated with [an] example image and the corresponding 
component of [the same type of] information item associated with [a] stored image under consideration", whereby a 
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similarity between the images is evaluated. [Smith97] also teaches histogram intersection as means for evaluating 
the similarity between images. Histogram intersection is also taught in [ChangSmith95]. 

3 1 . [FolkersOO]. [FolkersOO] discloses a method for processing pictorial queries into an image database. 
[FolkersOO] evaluates the similarity of a query image and a database image, with respect to three levels of similarity: 
the matching similarity level (msl), the contextual similarity level (csl), and the spatial similarity level (ssl) (refer to 
Algorithm 3. 1. 1 on page 16 of [FolkersOO]; see also Table 2.2 on page 1 2). [FolkersOO] uses graphs to represent the 
query image and the database images. Notice that the graph representation provides an implicit indication of the 
spatial distribution of image content within an image (this can be seen from, e.g., Fig. 2.8 on page 14 of [FolkersOO] 
and by taking into account the spatial constraints defined in Table 2.2). Images that are spatially similar to the query 
image have graph representations that are isomorphic to the graph representation of the query image, while adhering 
to the spatial constraints imposed by ssl. Thus, [FolkersOO] teaches evaluating the spatial similarity of images by 
determining whether an isomorphism exists between their respective graph representations. 

32. As discussed above, [ChangSmith95] stores the set of texture features (i.e. the "second information item") 
in a spatial quad-tree. It was pointed out earlier that the quad-tree is indicative of the spatial distribution of texture 
features- throughout the image. The previous Office Action contended that, because a quad-tree is a graph (as all 
trees are) representing an image, one of ordinary skill in the art would realize that the similarity between the query 
image and database images could be determined by an isomorphic graph matching algorithm, such as [FolkersOO]. 
Clearly, if an isomorphism exists between the quad-trees of two images, then the images can be assumed to have 
similar spatial distributions of whatever features are associated with the nodes (e.g. textural features). In this way, 
the isomorphism constraint reinforces the similarity between images by ensuring not only the similarity of texture 
features, but also a similarity in the spatial distribution of these features. 

Rejections Under 35 U.S.C. $ 102(b^ 

33. The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the 
rejections under this section made in this Office action: 
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(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in this 
country, more than one year prior to the date of application for patent in the United States. 

34. Claims 1-3, 5-7, 9-10. 17 24. 17-18. 20-24. and 31 are rejected under 35 U.S.C. § 102(b) as being 
anticipated by [ChangSmith95]. 

35. The following is in regard to Claim L [ChangSmith95] disclose method for extracting multi-dimensional 
features (e.g. color, texture shape, etc.) from images for the purposes ([ChangSmith95], Section 2: Extracting Low- 
Level Visual Features - Texture, Color, and Shape) of querying and indexing a database of images 
([ChangSmith95], Section 5, paragraph 1). The method operates on images comprising a plurality of blocks (e.g. 
blocks of belonging to quad-tree decompositions of the images - cf. [ChangSmith95] Fig. 4) comprises the 
following steps: 

(1 .a.) Generating a first information item (i.e. the set of color features or color histograms - 

[ChangSmith95] Section 2.2: Quad-Tree Based Color Histogram Indexing) characteristic 
of the visual content of the said image. Refer also to the discussion in the previous 
section of this document. 

(1 .b.) Generating a second information item (i.e. the set of texture features of the image - 

[ChangSmith95] Section 2.1 : Texture and Fig. 2). As illustrated above, the set of texture 
features is indicative of a degree of visual significance of the visual content of the blocks 
with respect to the overall visual content of the image, and, thereby, characteristic of the 
spatial distribution of the visual content of the image in its image plane. Please refer to 
the discussion of [ChangSmith95] in the previous section of this document. 

(1 .c.) Associating, with the said image, an index (i.e. an integrated feature map - 

[ChangSmith95] Section 2.3, last paragraph, and Section 3) composed of the said first 
information item and the said second information item. 



36. The following is in regard to Claim 2. The method of [ChangSmith95] further comprises: 
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(2.a.) Dividing the image plane of the said image according to a partitioning comprising the 
plurality of blocks, the plurality of blocks comprising a predefined number N of blocks. 
As discussed above, [ChangSmith95] perform a quad-tree decomposition on the image. 
Although not explicitly disclosed in [ChangSmith95], the decomposition must be subject 
to some predefined stopping criterion. Stopping according to feature (color or texture) 
homogeneity alone could result in excessively large tree. Typically, decomposition is 
stopped after a predefined number of partitions have been obtained or when the tree has 
grown to a certain depth. Such stopping criteria are inherent to any practical 
implementation of quad-tree decomposition. 
Generating the set of texture features involves: 

(2.b.) Extracting from each of the said blocks a data item of a first type (i.e. the histograms of 
each block of the decomposed image - [ChangSmith95] Section 2.2: Quad-Tree Based 
Color Histogram Indexing) representing at least one characteristic (i.e. color) of the 
visual content of the block under consideration. A discussion of these histograms was 
given above. Please refer to the discussion of [ChangSmith95] in the previous section of 
this document. 

(2.c.) Generating the said first information item as being a vector having N components, each 
of which is one of the said data items of the first type. Clearly, if a histogram is generated 
for each of the N blocks, then the set of block histograms consists of N elements. Note 
that a set is a mathematical abstraction and is typically represented, in practical 
computational systems, as a vector. Therefore, the set of N block histograms is, for all 
intents a purposes, a vector of N block histograms. 

37. The following is in regard to Claims 3, 5-7, 9-10, 17 24. 17-18. 19-24. and 31. The rejections of Claims 3, 
5-7, 9-10, 17-24, and 31 set forth in the previous Office Action are incorporated, herein, by reference. For the sake 
of brevity, the details of these rejections are omitted. Please refer to the appropriate sections of the previous Office 
Action, as well as the discussion of [ChangSmith95] above. 
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Rejections Under 35 U.S.C. $ 103(a) 

38. The following is a quotation of 35 U.S.C. § 103(a) which forms the basis for all obviousness rejections set 
forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if 
the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would 
have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

39. Claims IK 12, 14. 25, and 27-29 are rejected under 35 U.S.C. § 103(a) as being unpatentable over 
[ChangSmith95], in view of [SmithChang94] ; 

40. The following is in regard to Claims 11, 14, 25, and 27-29. The rejections of Claims 11, 14, 25, and 27-29 
set forth in the previous Office Action are incorporated, herein, by reference. For the sake of brevity, the details of 
these rejections are omitted. Please refer to the appropriate sections of the previous Office Action, as well as the 
discussion of [SmithChang94] above. 

41. The following is in regard to Claim 12. The basis of the rejection of Claim 12 was provided on page 10, 1 
24 of the Original Office Action. For the sake of clarity this will be reiterated. The teachings of rChangSmith951 can 
be combined with those of [SmithChang941 in such a way as to satisfy all the limitations of Claim 11. The step of 
calculating the first similarity in the method of fSmithChang94] is implemented by calculating a distance (e.g. 
Mahalanobis distance between textures features - Section 4.1 on page 4 of fSmithChang941) between the second 
information item associated with example image and the second information item associated with the said stored 
image under consideration. In this way, the teachings of lChangSmith95] and [SmithChang941. when combined in 
the manner described above, yield a method of searching for image that conforms to all the limitations of Claim 12. 

42. Claim 13 is rejected under 35 U.S.C. § 103(a) as being as being unpatentable over [ChangSmith95], in 
view of [FolkersOO]. 
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43. The following is in regard to Claim 13. The rejection of Claim 13 set forth in the previous Office Action is 
incorporated, herein, by reference. For the sake of brevity, the details of this rejections are omitted. Please refer to 
the appropriate sections of the previous Office Action, as well as the discussion of [FolkersOO] above. 

44. Claims 15-16 are rejected under 35 U.S.C. § 103(a) as being unpatentable over [ChangSmith95], in view of 
[Smith97]. 



45. The following is in regard to Claims 15-16. The rejections of Claims 15-16 set forth in the previous Office 
Action are incorporated, herein, by reference. For the sake of brevity, the details of these rejections are omitted. 
Please refer to the appropriate sections of the previous Office Action, as well as the discussion of [Smith97] above. 
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Allowable Subject Matter 

Objections, Allowable Subject Matter 

46. Claims 4, 8, 26, and 34- 30 are objected to as being dependent upon a rejected base claim, but would be 
allowable if rewritten in independent form including all of the limitations of the base claim and any intervening 
claims. 

Citation of Relevant Prior Art 

47. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: 

[SmithChTexture94] J.R. Smith and S. Chang, Transform Features for Texture Classification and 
Discrimination in Large Image Databases. IEEE ICIP-94, 1994. 

[VanDeWouwer97] G. Van de Wouwer, Color Texture Classification by Wavelet Energy 
Correlation Signatures. ICIAP (1): 327-334, 1997. 

[ChangKuo93] T. Chang and J. Kuo, Texture Analysis and Classification with Tree- 

Structured Wavelet Transform. IEEE Transactions on Image Processing, 
Vol. 2, No. 4, October 1993. 

[LiangKuo99] K. Liang and J. Kuo, WaveGuide: A Joint Wavelet-Based Image 

Representation and Description System. IEEE Transactions on Image 
Processing, Vol. 8, No. 1 1, November 1999. 

48. The following are particularly relevant CBIR systems: 

[Moghaddam99] B. Moghaddam et al., Defining Image Content with Multiple Regions-of Interest, 
IEEE Workshop on Content-Based Access of Image and Video Libraries, June 
1999. 
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[Yamamoto99] H. Yamamoto et al., Content-Based Similarity Retrieval of Images Based on 

Spatial Color Distributions , IEEE Proceedings of the International Conference 

on Image Analysis and Processing, September 1999. 
[Dupplaw99] D. Dupplaw et al., Spatial Colour Matching for Content Based Image Retrieval 

and Navigation, Challenge of Image Retrieval, 1999. 
[Malki99] J. Malki et al., Region Queries without Segmentation for Image Retrieval by 

Content, International Conference on Visual Information Systems (VISUAL'99), 

Lecture Notes in Computer Science vol. 1614, pp. 1 15-122., June 1999. 
[Yihong94] G. Yihong et al., An Image Database System with Fast Image Indexing 

Capability Based on Color Histograms, TENCON'94, August 1994. 
[Celentano98] A. Celentano and E. Di Sciascio, Feature Integration and Relevance Feedback 

Analysis in Image Similarity Evaluation, Journal of Electronic Imaging 7(2), pp. 

308-317, April 1998. 

49. The following relate to subgraph isomorphism and its application to CBIR and scene analysis: 
[FolkersOO] A. Folkers et al., Processing Pictorial Queries with Multiple Instances using 

Isomorphic Subgraphs, Proceedings of the 1 5th International Conference on Pattern 
Recognition, Vol. 4, No. 3-7, September 2000. 
[Ullmann] J.R. Ullmann, An Algorithm for Subgraph Isomorphism, JACM Vol. 23 , Issue 1 , 
January 1976. 

50. [Lee94] discloses a multi-resolution video indexing method which matches video frames by analyzing a 
multi-resolution decomposition of the video frames and hierarchically comparing local histograms at different 
resolutions. [Lee94] also discloses various metrics for evaluating the similarity of histograms. 

[Lee94] J. Lee and B. Dickinson, Multiresolution Video Indexing For Subband Coded Video 

Databases, Proceedings of IS&T/SPIE, Conference on Storage and Retrieval for Image 
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and Video Databases, 1994. 



51. [Huang97] applies the concept of correllograms to CBIR. A Correllogram is a single feature embedding 
both spatial correlation and color information. 

[Huang97] J. Huang et al., Image Indexing Using Color Correlograms, Proceedings of the IEEE 
Computer Society Conference on Computer Vision and Pattern Recognition, 1997. 

52. Other relevant literature by S. Chang and J.R. Smith related to the VisualSEEk system. 
[SmithChTooIs96] J.R. Smith and S. Chang, Tools and Techniques for Color Image Retrieval, 

IS&T/SPIE Proceedings Vol. 2670, Storage & Retrieval for Image and Video 
Databases IV, 1996. 

[SmithChLocal96] J.R. Smith and S. Chang, Local Color and Texture Extraction and Spatial 
Query, Proceedings of the International Conference on Image Processing, 
1996. 

[SmithChVisual96] J.R. Smith and S. Chang, VisualSEEk: A Fully Automated Content-Based 
Image Query System, ACM Multimedia '96, 1 996. 

53. Other commercial and academic CBIR systems: 

[Virage96] J.R. Bach et al., The Virage Image Search Engine: An Open Framework for 



Computer Magazine, September 1995. 

[Surflmage98] C. Nastar et al., Surflmage: A Flexible Content-Based Image Retrieval System. 
ACM Multimedia '98, 1998. 

[ImageRover97] S. Sclaroff et ah, ImageRover: A Content-Based Image Browser for the World- 
Wide Web, Proceedings of the IEEE Workshop on Content-Based Access of 



Image Management, SPIE Vol. 2670, March 1996. 



[QBIC95] 



Flickner et al., Query by Image and Video Content: The QBIC System, IEEE 
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Image and Video Libraries, June 1997. 

54. Other CBIR literature related to the evaluation and comparison of global and local image features, spatial 
distributions of image features, etc. 

[Milanese96] R. Milanese et al., Correspondence Analysis and Hierarchical Indexing for 

Content-Based Image Retrieval, Proceedings of the International Conference on 
Image Processing, 1996. 

[Manmatha98] R. Manmatha et al., On Computing Local and Global Similarity in Images, 

Technical Report MM-25, Center for Intelligent Information Retrieval, Computer 
Science Dept., University of Massachusetts, Amherst, Massachusetts, 1998. 

[Jain96] A, Jain et al., Image Retrieval Using Color and Shape, Pattern Recognition 29 (8), 

pp. 1233-1244, 1996. 

[DelBimbo98] A, Del Bimbo et al., Using Weighted Spatial Relationships in Retrieval by Visual 
Contents, Proceedings of the IEEE Workshop on Content - Based 
Access of Image and Video Libraries, 1998. 

55. A fairly comprehensive survey of CBIR systems circa 2002. 

[Veltkamp02] R. Veltkamp et al., Content-Based Image Retrieval Systems: A Survey, October 
2002. 



Conclusion 

56. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 
37CFR 1.136(a). 
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57. A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the 
mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final 
action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, 
then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee 
pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, 
will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. 
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Any inquiry concerning this communication or earlier communications from the examiner should be 
directed to Kevin Siangchin whose telephone number is (703)305-7569. The examiner can normally be reached on 
9:00am - 5:30pm, Monday - Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Amelia Au can 
be reached on (703)308-6604. The fax phone number for the organization where this application or proceeding is 
assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent Application Information 
Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR 
or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more 
information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the 
Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

Kevin Siangchin 
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